[exporter/elasticsearch] add dynamic document pipeline support for logs #37860
Conversation
| @@ -145,6 +145,9 @@ This can be customised through the following settings: | |||
| - `logs_dynamic_id` (optional): Dynamically determines the document ID to be used in Elasticsearch based on a log record attribute. | |||
| - `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document. See [Setting a document id dynamically](#setting-a-document-id-dynamically). | |||
|
|
|||
| - `logs_dynamic_pipeline` (optional): Dynamically determines the ingest pipeline to be used in Elasticsearch based on a log record attribute. | |||
| - `enabled`(default=false): Enable/Disable dynamic pipeline for log records. If `elasticsearch.document_pipeline` exists and is not an empty string in the log record attributes, it will be used as the Elasticsearch ingest pipeline. The attribute `elasticsearch.document_pipeline` is removed from the final document. | |||
There was a problem hiding this comment.
The attribute
elasticsearch.document_pipelineis removed from the final document.
nit: While this is true for OTel mode, it doesn't seem to be the case for other modes. I'm fine with that, but we should state that in the documentation. (I think I either missed this for elasticsearch.document_id in its PR review or something has changed since then)
- Added caveat that the attributes are only removed when `otel` mapping mode is used
leehinman
force-pushed
the
37419_elasticsearch_pipeline_id
branch
from
642ff7f to
8ef035c
Compare
|
@open-telemetry/collector-contrib-maintainers can someone please approve the workflow? |
jpkrohling
changed the title
| @@ -188,6 +191,7 @@ Documents may be optionally passed through an [Elasticsearch Ingest pipeline] pr | |||
| This can be configured through the following settings: | |||
|
|
|||
| - `pipeline` (optional): ID of an [Elasticsearch Ingest pipeline] used for processing documents published by the exporter. | |||
| - If `elasticsearch.document_pipeline` exists and is not an empty string in the log record attributes, then that pipeline will be used for that log record. | |||
There was a problem hiding this comment.
Shouldn't this mention that this works when the logs_dynamic_pipeline::enabled property is set to true?
| - `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document. See [Setting a document id dynamically](#setting-a-document-id-dynamically). | ||
| - `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document when the `otel` mapping mode is used. See [Setting a document id dynamically](#setting-a-document-id-dynamically). | ||
|
|
||
| - `logs_dynamic_pipeline` (optional): Dynamically determines the ingest pipeline to be used in Elasticsearch based on a log record attribute. |
There was a problem hiding this comment.
[question] I wonder why this is a logs-specific setting. Would it make sense to make this available to other signals as well - metrics, traces, maybe profiles?
There was a problem hiding this comment.
I agree. We can make it available for all of them. I don't think this is much of a work.
On a side note, I think we should do same for dynamic document id setting which was recently introduced. It is only supported for logs for now. But I think we can make it available for all signals (in another PR ofcourse)
Description
This PR adds a new config option
logs_dynamic_pipelinethat when set totrue reads the
elasticsearch.document_pipelineattribute from each logrecord and uses it as the ingest pipeline in Elasticsearch. This is
only implemented for logs, but a subsequent PR supporting metrics and
traces could be opened.
Link to tracking issue
Fixes #37419
Testing
Added tests to verify that the document pipeline attribute can be read
from the log record and that the pipeline is properly forwarded to
Elasticsearch. Also asserted that when there is no document pipeline
attribute the current behavior is retained.
Manual testing that setting the option resulted in the named ingest
pipeline being run on the document in Elasticsearch.
Documentation
Updated the readme to mention the new
logs_dynamic_pipelineconfig option.